Crowdsourcing for Social Multimedia at MediaEval 2013: Challenges, Data set, and Evaluation

This paper provides an overview of the Crowdsourcing for Multimedia Task at MediaEval 2013 multimedia benchmarking initiative. The main goal of this task is to assess the potential of hybrid human/conventional computation techniques to generate accurate labels for social multimedia content. The task data are fashion-related images, collected from the Web-based photo sharing platform Flickr. Each image is accompanied by a) its metadata (e.g., title, description, and tags), and b) a set of ‘basic human labels’ collected from human annotators using a microtask with a basic quality control mechanism that is run on the Amazon Mechanical Turk crowdsourcing platform. The labels reflect whether or not the image depicts fashion, and whether or not the image matches its ‘category’ (i.e., the fashion-related query that returned the image from Flickr). The ‘basic human labels’ were collected such that their noise levels would be characteristic of data gathered from crowdsourcing workers without using highly sophisticated quality control. The task asks participants to predict high-quality labels, either by aggregating the ‘basic human labels’ or by combining them with the context (i.e., the metadata) and/or the content (i.e., visual features) of the image.